Imaging system and methods displaying a fused multidimensional reconstructed image

ABSTRACT

In some embodiments, an imaging system displays a multidimensional visualization of a surgical site. In some embodiments, the imaging system receives a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site. At the selected portion of the multidimensional visualization, the imaging system displays a portion of a multidimensional reconstructed image which corresponds to the selected multidimensional visualization such that the displayed portion of the multidimensional reconstructed image is fused with the displayed multidimensional visualization.

PRIORITY CLAIM

The present application claims priority to and the benefit of PCT PatentApplication No. PCT/US2013/057686, filed on Aug. 30, 2013, which claimspriority to and the benefit of U.S. Provisional Patent Application No.61/695,230, filed on Aug. 30, 2012, the entirety of which areincorporated herein by reference.

BACKGROUND

Brain and other organ surgery involve complex surgical procedures toaccess intricate and delicate portions of tissue. Oftentimes, surgeonswill image areas of a patient's body where the surgery is to beperformed. These images help surgeons plan how the surgery is to beperformed, identify specific areas of tissue that need to be accessed,and determine pathways through the body for surgical tools and camerasto access the target tissue.

In a typical surgery, surgeons will generally first image an area of apatient where the surgery is to be performed. Surgeons will meticulouslyreview these images to plan how the surgery is to be preformed. Evenduring surgery, surgeons may again review physical copies of theseimages or access a video monitor and scroll through the images as a wayto refresh their memory or to help determine their bearings. An issuewith this procedure is that it requires surgeons to look at the surgicalsite, then direct their attention to a video monitor or physical images,and then redirect their attention back to the patient. In other words,the surgeons have to mentally relate the images to the anatomy of thepatient.

This diversion of attention between patient and images may be mentallytaxing on a surgeon during a relatively long surgery. This may alsoextend the length of a surgery if a surgeon has to refer to the imagesmany times. Further, this may be especially tricky and time consumingfor a surgeon when an orientation of the images does not match up to thesurgeon's current view of the patient. For instance, a set of images ofan MRI scan of a patient's head may include hundreds of individualimages layered in a straight and level orientation. A surgeon lookingdown at the top of the patient's head to determine where to make anincision to reach a deeply embedded tumor has to construct and rectifyin his mind: (1) the different layers of MRI images between the top ofthe head and the level of the tumor, (2) the orientation of the MRIimages versus the orientation of the patient, and (3) the specificlocation on the MRI images as corresponding to an actual location on thepatient.

To increase the accuracy of incisions made during surgery and todecrease the amount of surgery time, it is desirable to provide surgeonswith new types of imaging systems. Accordingly, a need exits for furtherdevelopment of imaging systems.

SUMMARY

In some embodiments, the imaging systems, imaging apparatuses andimaging methods fuse portions of a multidimensional reconstructed imagewith multidimensional visualizations of at least a portion of a surgicalsite. The imaging systems may generate multidimensional reconstructedimages based on pre-operative image data. At a selected portion of thevisualization, the imaging systems may display a portion of themultidimensional reconstructed image.

In some embodiments, imaging systems display a window through a livesurgery visualization into a multidimensional reconstructed image belowa surface of at least a portion of a surgical site. Such a configurationmay be referred to as providing “x-ray vision” window capability.

In some embodiments, imaging systems enable users to control the windowto suit their immediate needs. For example, using a mouse, joystick,foot pedal, or any other suitable control device, imaging systems mayenable a user to control the window in x, y, and z directions. Suchcontrol devices also may enable a user to control the orientation of thedisplayed portion of the multidimensional reconstructed image. Forexample, imaging systems may enable the user to control the orientationsvia a yaw button, pitch button and roll button. Still further, imagingsystems may enable a user to control the scale or transparency of thedisplayed multidimensional reconstructed image.

The window may have any suitable shape. The window may be round, square,rectangular, elliptical, or any other geometric shape. In someembodiments, the window may also have an anatomical shape (e.g., thewindow may follow the outline of a tumor or organ). In some embodiments,the window is fused or blended with the live multidimensionalvisualization at the edges using a fading level of transparency (e.g.,alpha blending). In some embodiments, the overall window itself may bealpha blended with the live multidimensional visualization usingadjustable transparency.

In some embodiments, imaging systems enable a user to adjust the windowposition and shape. In some embodiments, the imaging system mayalgorithmically drive the imaging system to follow notations orhighlighted anatomy throughout the course of a surgical procedure.

Additional features and advantages are described herein and will beapparent from the following Detailed Description and the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example network communicating system,according to an example embodiment of the system disclosed herein.

FIG. 2 is a block diagram showing an example of a computing device,according to an example embodiment of the system disclosed herein.

FIG. 3 is a block diagram of an example structure in accordance with anexample embodiment of the imaging system disclosed herein.

FIG. 4 is a flowchart illustrating an example procedure for displaying aportion of a multidimensional reconstructed image at a selected portionof a multidimensional visualization.

FIG. 5 is a front view of an example imaging system, illustrating aportion of an example multidimensional reconstructed image being fusedwith a multidimensional visualization of a patient's head.

FIGS. 6 and 7 are front views of an example imaging system, illustratinga portion of an example multidimensional reconstructed image beingoverlayed on a displayed multidimensional visualization of a patient'shead.

FIG. 8 is perspective view of an example imaging system, illustratingthe imaging system being retrofitted onto a surgical device.

FIG. 9 is a block diagram showing an example data architecture,according to an example embodiment of the imaging system disclosedherein.

DETAILED DESCRIPTION

The present disclosure relates in general to systems for displaying aportion of a multidimensional image at a selected portion of avisualization of a surgical site. Imaging systems enable user(s) toselect the portion of the visualization in which a portion of amultidimensional reconstructed image is displayed.

The imaging systems described herein may be readily realized in anetwork communications system. A block diagram of an example networkcommunications system 10 (“system 10”) is illustrated in FIG. 1. In thisexample, system 10 includes an imaging system 12 and server 14.

It should be appreciated that users as described herein may include anyperson or entity which uses the presently disclosed system and mayinclude a wide variety of parties. For example, reference is made hereinto surgeons operating surgical equipment including the exemplary imagingsystems. It should be appreciated that the exemplary imaging system maybe used by anyone including nurses, surgical assistants, veterinarypersonnel, autopsy technicians, medical students, surgical residents,the surgeon's staff or anyone else who is to view a portion of amultidimensional reconstructed image being fused with a visualization ofa surgical site.

Imaging system 12 and/or server 14 may be configured according to itsparticular operating system, applications, memory, hardware, etc., andmay provide various options for managing the execution of the programsand applications, as well as various administrative tasks. Imagingsystem 12 and/or server 14 may interact via at least one network with atleast one other imaging system 12 and/or server 14, which may beoperated independently. Information processing systems 12 and servers 14operated by separate and distinct entities may interact togetheraccording to some agreed upon protocol.

A detailed block diagram of the electrical systems of an examplecomputing device is illustrated in FIG. 2. The example computing devicemay include any of the devices and systems described herein, includingimaging system 12 and server 14. In this example, the example computingdevices may include main unit 20 which preferably includes at least oneprocessor 22 electrically connected by address/data bus 24 to at leastone memory device 26, other computer circuitry 28, and at least oneinterface circuit 30. Processor 22 may be any suitable processor, suchas a microprocessor from the INTEL® CORE® family of microprocessors.Processor 22 may include one or more microprocessors, central processingunits (CPUs), computing devices, microcontrollers, graphics processingunits (GPUs), digital signal processors, or like devices or anycombination thereof. Memory 26 preferably includes volatile memory andnon-volatile memory. Preferably, memory 26 stores software program(s) orinstructions that interact with the other devices in system 10 asdescribed below. This program may be executed by processor 22 in anysuitable manner. In an example embodiment, memory 26 may be part of a“cloud” such that cloud computing may be utilized by imaging system 12and server 14. Memory 26 may also store digital data indicative ofimages, documents, files, programs, web pages, etc. retrieved fromcomputing devices 12, 14 and/or loaded via input device 32.

Interface circuit 30 may be implemented using any suitable interfacestandard, such as an Ethernet interface and/or a Universal Serial Bus(USB) interface. At least one input device 32 may be connected tointerface circuit 30 for entering data and commands into main unit 20.For example, input device 32 may be at least one of a keyboard, mouse,joystick, touch screen device, remote control, foot-pedal device,gesture recognition device, track pad, track ball, isopoint, characterrecognition, barcode scanner, and a voice recognition system. In oneexample embodiment, at least one input device 32 includes an imagesensor and/or camera system, such as photosensor 33.

As illustrated in FIG. 2, at least one display device 34, printers,speakers, and/or other output devices 36 may also be connected to mainunit 20 via interface circuit 30. Display device 34 may be any devicecapable of displaying a still or video image. Preferably, display device34 displays high definition (HD) still images and video images or videoswhich provide a surgeon with a greater level of detail than a standarddefinition (SD) signal. More preferably, display device 34 is configuredto display HD stills and images in three dimensions (3D). Exemplarydisplay devices include HD monitors, cathode ray tubes, projectionscreens, liquid crystal displays, organic light emitting diode displays,plasma display panels, light emitting diodes, 3D equivalents thereof andthe like. In some embodiments, display device 34 includes a 3D HDholographic display system. In one example embodiment, display device 34is a projection cart display system and incorporates the basicstructural components of the Applicant's TrueVision Systems, Inc.stereoscopic image display cart described in the Applicant's co-pendingU.S. application Ser. No. 11/739,042, entitled “Stereoscopic DisplayCart and System” filed Apr. 23, 2007, which is fully incorporated hereinby reference as if part of this specification. For example, displaydevice 34 may provide a user interface, which will be described infurther detail below, and may display at least one web page receivedfrom imaging system 12 and/or server 14. A user interface may includeprompts for human input from user 16 including links, buttons, tabs,checkboxes, thumbnails, text fields, drop down boxes, etc., and mayprovide various outputs in response to the user inputs, such as text,still images, videos, audio, and animations.

At least one storage device 38 may also be connected to main device orunit 20 via interface circuit 30. At least one storage device 38 mayinclude at least one of a hard drive, CD drive, DVD drive, and otherstorage devices. At least one storage device 38 may store any type ofdata, such as multidimensional visualization data, multidimensionalreconstructed image data, selection data, window data, image data,content data, statistical data, historical data, databases, programs,files, libraries, and/or other data, etc., which may be used by imagingsystem 12 and/or server 14.

Imaging system 12 and/or server 14 may also exchange data with othernetwork devices 40 via a connection to network 18. Network devices 40may include at least one server 42, which may be used to store certaintypes of data, and particularly large volumes of data which may bestored in at least one data repository 44. Server 42 may include anykind of data 46 including multidimensional visualization data,multidimensional reconstructed image data, selection data, window data,image data, content data, statistical data, historical data, databases,programs, files, libraries, and/or other data, etc. Server 42 may storeand operate various applications relating to receiving, transmitting,processing, and storing the large volumes of data. It should beappreciated that various configurations of at least one server 42 may beused to support and maintain system 10. In some example embodiments,server 42 is operated by various different entities, including privateindividuals, administrative users and/or commercial partners. Also,certain data may be stored in imaging system 12 and/or server 14 whichis also stored on server 42, either temporarily or permanently, forexample in memory 26 or storage device 38. The network connection may beany type of network connection, such as an ethernet connection, digitalsubscriber line (DSL), telephone line, coaxial cable, wirelessconnection, etc.

Access to imaging system 12 and/or server 14 can be controlled byappropriate security software or security measures. A user's access canbe denied to imaging system 12 and/or server 14 and be limited tocertain data and/or actions. Accordingly, users of system 10 may berequired to register with imaging system 12 and/or server 14.

As noted previously, various options for managing data located withinimaging system 12, server 14 and/or in server 42 may be implemented. Amanagement system may manage security of data and accomplish varioustasks such as facilitating a data backup process. The management systemmay update, store, and back up data locally and/or remotely. Amanagement system may remotely store data using any suitable method ofdata transmission, such as via the Internet and/or other networks 18.

FIG. 3 is a block diagram showing an example imaging system 300. Imagingsystem 300 is operated by a user such as a surgeon. It should beappreciated that imaging system 300 illustrated in FIG. 3 may beimplemented as imaging system 12.

As illustrated in FIG. 3, in this example, imaging system 300 includesdatabase system 302, multidimensional visualization generation system304, multidimensional image reconstruction module 306, window generationmodule 308, fusion module 310, multidimensional reconstructed imagedepth adjustment module 312, multidimensional reconstructed imagehighlight module 314, and multidimensional reconstructed image filtermodule 316. Database system 302, multidimensional visualizationgeneration system 304, multidimensional image reconstruction module 306,window generation module 308, fusion module 310, multidimensionalreconstructed image depth adjustment module 312, multidimensionalreconstructed image highlight module 314, and multidimensionalreconstructed image filter module 316 may include software and/orhardware components, such as a field programmable gate array (FPGA) oran application specific integrated circuit (ASIC), which performscertain tasks. Database system 302, multidimensional visualizationgeneration system 304, multidimensional image reconstruction module 306,window generation module 308, fusion module 310, multidimensionalreconstructed image depth adjustment module 312, multidimensionalreconstructed image highlight module 314, and multidimensionalreconstructed image filter module 316 may advantageously be configuredto reside on an addressable storage medium and configured to be executedon one or more processors. Thus, database system 302, multidimensionalvisualization generation system 304, multidimensional imagereconstruction module 306, window generation module 308, fusion module310, multidimensional reconstructed image depth adjustment module 312,multidimensional reconstructed image highlight module 314, andmultidimensional reconstructed image filter module 316 may include, byway of example, components, such as software components, object-orientedsoftware components, class components and task components, processes,functions, attributes, procedures, subroutines, segments of programcode, drivers, firmware, microcode, circuitry, data, databases, datastructures, tables, arrays, and variables. The functionality providedfor in the components and modules may be combined into fewer componentsand modules or further separated into additional components and modules.

Database system 302 may include a wide variety of data. For example,database system 302 may include any of the following data:multidimensional visualization data, patient data, multidimensionalreconstructed image data, selection data, window data, image data,content data, statistical data, historical data, databases, programs,files, libraries, and/or other data, etc.

Database 302 may receive any of the above-mentioned data from a hospitalinformation system.

In some embodiments, multidimensional visualization generation system304 generates and displays multidimensional visualizations of at least aportion of a target surgical site. The multidimensional visualizationsmay include images and/or videos and are preferably in 3D and HD.Multidimensional visualization generation system 304 may generatevisualizations using a photosensor. The photosensor may respond to anyor all of the wavelengths of light that form the electromagneticspectrum. Alternatively, the photosensor may be sensitive to a morerestricted range of wavelengths including at least one wavelength oflight outside of the wavelengths of visible light. Visible light mayrefer to light having wavelengths corresponding to the visible spectrum,which is that portion of the electromagnetic spectrum where the lighthas a wavelength ranging from about 380 nanometers (nm) to about 750 nm.

In some embodiments, multidimensional image reconstruction module 306generates a multidimensional reconstructed image. Multidimensional imagereconstruction module 306 preferably generates 3D images using 2Dpre-operative or intra-operative image slices of a surgical area of apatient. In some example embodiments, multidimensional imagereconstruction module 306 generates 3D images using vector-based orfeature based construction to create outlines or boundaries of 3Dobjects from sequential pre-operative 2D image slices.

In some embodiments, window generation module 308 generates and displaysa window. Window generation module 308 may generate and display a windowbased on selection which corresponds to a portion of the generatedmultidimensional visualization. In some embodiments, window generationmodule 308 enables a user to adjust the size of the displayed window.

In some example embodiments, fusion module 310 fuses, merges, blends,joins or integrates a multidimensional reconstructed image with adisplayed multidimensional visualization. Fusion module 310 may fuse themultidimensional reconstructed image with a multidimensionalvisualization based on the orientation or coordinates of the surgicalsite in the multidimensional visualization. In some embodiments, fusionmodule 310 fuses the multidimensional reconstructed image with amultidimensional visualization based on pattern matching by identifying3D features, structures, or objects in the multidimensionalvisualization with corresponding features, structures, or objects in themultidimensional reconstructed image. In some embodiments, fusion module310 combines appropriate portions of a video signal of themultidimensional visualization with the selected portion of themultidimensional reconstructed image. In some embodiments, fusion module310 transmits the video signal of the multidimensional visualization ofthe surgical site to the display device separately from themultidimensional reconstructed image. In these embodiments, fusionmodule 310 sends an instruction to the display device to display themultidimensional reconstructed image as a graphic atop the appropriateportion of the multidimensional visualization. The instruction may besent using an HD multimedia interface (“HDMI”) communication protocol.In some embodiments, fusion module 310 combines video of themultidimensional visualization with a multidimensional reconstructedimage by adjusting visual properties of the multidimensionalvisualization so as to make the multidimensional visualization merge orfuse with the multidimensional reconstructed image. Adjusting visualproperties of the multidimensional visualization may include: (a)increasing the transparency of a portion of a multidimensionalvisualization so that a multidimensional reconstructed image replacesthat portion of the multidimensional visualization; and (b) adjustingvisual properties such as adjusting contrast and focus or applying aspline function around edges or fringes where the multidimensionalreconstructed image borders a multidimensional visualization. In someembodiments, an image guidance system (“IGS”) device registers themicroscope with the patient and places the microscope field of view intothe pre-operative imaging scan data passing all of the coordinates toour device via Ethernet connection. In some embodiments, imaging system12 employs machine vision algorithms to identify a target structure(such as a specially marked screw placed into the bones of the spine).The target structure may be registered intra-operatively using an O-armor C-arm imaging device. The image data from the O-arm or C-arm can thenbe communicated into our system and fused with the pre-operative dataand the live surgical view.

In some embodiments, multidimensional reconstructed image depthadjustment module 312 receives requests to display a portion of amultidimensional reconstructed image having a certain depth.Multidimensional reconstructed image depth adjustment module 312 mayenable a user to increase or decrease the depth of the currentlydisplayed multidimensional reconstructed image.

In some embodiments, multidimensional reconstructed image highlightmodule 314 highlights certain features or portions of the displayedmultidimensional reconstructed image. Such features or portions mayinclude internal anatomical structures such as an aneurysm, a tumor orblood vessels.

In some embodiments, multidimensional reconstructed image filter module316 filters or enables a user to select and remove certain types ofanatomical structures of a multidimensional reconstructed image. Forexample, in one embodiment, imaging system 12 enables a user to selectto view only bone structures, brain tissue, blood vessels, tumors oraneurisms. Such a configuration enables users to focus themultidimensional reconstructed image on desired anatomical structuresthat are important for a surgery.

Although the above has been shown using imaging system 300, there can bemany alternatives, modifications, and variations. For example, some ofthe modules of the imaging system may be expanded and/or combined.Further, in some example embodiments, the functions provided by certainmodules may be employed by a separate imaging system operated by aseparate entity. In one example, imaging system 300 does not includedatabase system 302. In this example, imaging system 300 may beconfigured to communicate with a separate database system which includesthe data described in database system 302 shown in FIG. 3. Other systemsmay be inserted to those noted above. Depending upon the embodiment,database system 302, multidimensional visualization generation system304, multidimensional image reconstruction module 306, window generationmodule 308, fusion module 310, multidimensional reconstructed imagedepth adjustment module 312, multidimensional reconstructed imagehighlight module 314, and multidimensional reconstructed image filtermodule 316 may be replaced. Further details of these systems are foundthroughout the present specification.

Imaging system 300 may process data received from other devices. Forexample, another computing device (e.g., a personal computer) may querydata from database system 310 for use in a report.

Numerous embodiments are described in the present application, and arepresented for illustrative purposes only. The described embodiments arenot, and are not intended to be, limiting in any sense. The presentdisclosure may be widely applicable to numerous embodiments, as isreadily apparent from the disclosure. One of ordinary skill in the artwill recognize that the disclosure may be practiced with variousmodifications and alterations, such as structural, logical, software,and electrical modifications. Although particular features of thedisclosure may be described with reference to one or more particularembodiments and/or drawings, it should be understood that such featuresare not limited to usage in the one or more particular embodiments ordrawings with reference to which they are described, unless expresslyspecified otherwise.

As illustrated in FIG. 4, a flowchart of an example process 400 includesdisplaying a portion of a multidimensional reconstructed image at aselected portion of a multidimensional visualization of a surgical site.Preferably, process 400 is embodied in one or more software programswhich are stored in one or more memories and executed by one or moreprocessors. Although process 400 is described with reference to theflowchart illustrated in FIG. 4, it should be appreciated that manyother methods of performing the acts associated with process 400 may beused. For example, the order of the steps may be changed, some of thesteps described may be optional, and additional steps may be included.

More specifically, as indicated by block 402, imaging system 12 displaysa multidimensional visualization of a surgical site. In someembodiments, imaging system 12 includes a stereoscopic microsurgicalvisualization system (e.g., a camera or surgical microscope) to capturethe multidimensional visualization. The multidimensional visualizationis preferably a stereoscopic 3D real time video; however, it may be 2D.The use of a 3D visualization is preferred as it provides many benefitsto a surgeon including more effective visualization and depth of field.The multidimensional visualization may be referred to as a real timevideo.

As indicated by block 404, imaging system 12 may generate amultidimensional reconstructed image based on image data. The image datamay be at least one of pre-operative data and intra-operative data. Inone example embodiment, imaging system 12 generates a 3D image (ormodel) based on two-dimensional image slices by determining the sequenceof the two-dimensional images, identifying common structures between theimages, and forming corresponding three dimensional shapes. Imagingsystem 12 may generate the multidimensional reconstructed image in realtime as visualizations are generated. The multidimensional reconstructedimage may be at least one of a 3D multidimensional reconstructed image,a stereoscopic image, and a high definition “HD” image.

As indicated by block 406, imaging system 12 receives a selection whichcorresponds to a portion of the displayed multidimensional visualizationof the surgical site. In one example embodiment, imaging system 12receives the selection based on a user operating with an input device toselect or place a window within a portion of a displayed visualization.

As indicated by block 408, at the selected portion of themultidimensional visualization, imaging system 12 displays a portion ofthe multidimensional reconstructed image.

Referring to FIG. 5, this front view of an example imaging systemgenerally shows an example illustrating a portion of an examplemultidimensional reconstructed image being fused with a multidimensionalvisualization of a patient's head. In this embodiment, imaging system 12highlights a feature (i.e., a brain tumor) of the multidimensionalreconstructed image. In this example, surgeon's hand 505 grips cuttingtool 507 to cut through the patient's cranium 532. During an operationto remove brain tumor 530, imaging system 12 enables a surgeon to ‘see’where tumor 530 is located relative to adjacent patient anatomy andrelative to the features of the patient's head shown in visualization502.

In some embodiments, in response to a second selection which correspondsto a second portion of the visualization, imaging system 12 selects anddisplays a second portion of the multidimensional reconstructed image atthe second selected portion of the visualization.

In some embodiments, imaging system 12 displays a secondmultidimensional reconstructed image based on a change in a location,size or shape of the window.

In some embodiments, imaging system 12 receives a request selectionwhich corresponds to a different portion of the multidimensionalvisualization. For example, imaging system 12 may receive a selectionbased on a user selection of a move button. Referring to FIG. 5, imagingsystem 12 displays the following move buttons: up button 514, downbutton 516, left button 518 and right button 520. In response to a userselecting up button 514, imaging system 12 enables the user to cause theposition of window 503 to move up relative to the displayedmultidimensional visualization 502. It should be appreciated that whenwindow 503 is moved relative to the multidimensional visualization 502,imaging system 12 displays a different portion of the multidimensionalreconstructed image. In response to a user selecting down button 516,imaging system 12 enables the user to cause the position of window 503to move down relative to the displayed multidimensional visualization502. In response to a user selecting left button 518, imaging system 12enables the user to cause the position of window 503 to move leftrelative to the displayed multidimensional visualization 502. Inresponse to a user selecting right button 520, imaging system 12 enablesthe user to cause the position of window 503 to move right relative tothe displayed multidimensional visualization 502.

In some embodiments, in response to a change in the displayedmultidimensional visualization, imaging system 12 displays a differentportion of the multidimensional reconstructed image that corresponds tothe changed multidimensional visualization. Imaging system 12 maydetermine a change in the visualization has occurred based on at leastone of: (a) selections from input device 32; (b) indications from animage guidance system (“IGS”) that a surgical microscope has moved; (c)a detected difference between a current visualization and a previousvisualization.

In some embodiments, imaging system 12 enables a user to operate with aninput device to change the scale of the displayed multidimensionalreconstructed image. For example, referring to FIG. 5, in thisembodiment, imaging system 12 displays scale increase button 522 andscale decrease button 524. In response to a user selecting scaleincrease button 522, imaging system 12 enables the user to cause anincrease in the display size of at least one of: (a) the displayedmultidimensional reconstructed image; and (b) the displayedmultidimensional visualization. In response to a user selecting scaledecrease button 524, imaging system 12 enables the user to cause adecrease in the display size of at least one of: (a) the displayedmultidimensional reconstructed image; and (b) the displayedmultidimensional visualization.

In some embodiments, imaging system 12 displays a depth meter thatindicates a depth associated with the currently displayedmultidimensional reconstructed image. As a result, imaging system 12enables a user to view relatively deep patient anatomy before the useris able to visually see this anatomy in the displayed multidimensionalvisualization, via the naked eye, or through optical magnified viewingthrough a microscope or with surgical loupes. Referring to FIG. 5, inthis example, imaging system 12 displays depth meter 506 which indicatesa depth coordinate of the currently displayed multidimensionalreconstructed image. In this example, depth meter 506 indicates that thecurrently displayed multidimensional reconstructed image is beingdisplayed at a depth level of 5 cm.

Imaging system 12 also displays depth increase button 526 and depthdecrease button 528. In response to a user selecting depth increasebutton 526, imaging system 12 enables the user to cause an increase inthe viewing depth of the currently displayed multidimensionalreconstructed image. That is, in response to a user selecting depthincrease button 526, imaging system 12 displays a different portion ofthe multidimensional reconstructed image having an increased depth. Inresponse to a user selecting depth decrease button 528, imaging system12 enables the user to cause a decrease in the depth of the currentlydisplayed multidimensional reconstructed image. That is, in response toa user selecting depth decrease button 528, imaging system 12 displays adifferent portion of the multidimensional reconstructed image having adecreased depth.

In some embodiments, imaging system 12 may highlight certain features orportions of the displayed multidimensional reconstructed image. Suchfeatures or portions may include internal anatomical structures such asan aneurysm, a tumor or blood vessels. For example, referring to FIG. 5,in this example, imaging system 12 highlights tumor 530. As a result, inthis example, imaging system 12 enables a surgeon to ‘see’ where tumor530 is located relative to adjacent brain tissue and relative to thefeatures of patient's head shown in visualization 502. Such aconfiguration allows attention to be brought to important anatomicalstructures within the multidimensional reconstructed image.

Such features or portions may be highlighted in any suitable way. Forexample, a tumor may be highlighted with pseudo colors such as purpleand blood vessels may be highlighted with a second, different color.

The highlighted anatomy may be manually or automatically selectedpre-operatively in the imaging data or selected intra-operatively in thelive surgical view based on specific criteria in the image or data. Forexample, the software may select a specific range of Hounsfield Units orCT densities to identify a tumor or aneurism with calcification as thehighlighted anatomy.

In some embodiments, the imaging system inverts the imagingoverlay/underlay modality. In typical usage, the “x-ray window” willopen up a view that underlays the live surgical view and shows a smallportion of the multidimensional reconstruction within the boundaries ofthe live surgical multidimensional visualization.

Referring to FIG. 5, in this example embodiment, the portion ofmultidimensional reconstructed image 502 that is displayed within window503 is displayed as being underlayed with respect to the displayedmultidimensional visualization 502. That is, in this example, if thesurgeon were to move his thumb 505 over the currently displayed portionof the multidimensional reconstructed image 504, the surgeon's thumbwould block the view of the currently displayed multidimensionalreconstructed image.

FIG. 6 illustrates another example embodiment of an imaging system 12,illustrating multidimensional reconstructed image 602 being displayed asbeing underlayed with respect to the displayed multidimensionalvisualization 604. That is, the surgeon's thumb blocks the view of themultidimensional reconstructed image 602.

In an alternative embodiment, the portion of multidimensionalreconstructed image that is displayed within the window may be displayedas being overlayed with respect to the displayed multidimensionalvisualization. For example, referring to FIG. 7, the portion ofmultidimensional reconstructed image 702 is positioned over surgeon'sthumb 505 and is shown as being overlayed with respect tomultidimensional visualization 704. That is, unlike the multidimensionalreconstructed image shown in FIG. 5, the thumb does not block the viewof the multidimensional reconstructed image.

Imaging system 12 may fuse or merge the portion of the multidimensionalreconstructed image with the multidimensional visualization in anysuitable way. In one example embodiment, imaging system 12 fuses themultidimensional reconstructed image with the multidimensionalvisualization of a target site by using visual tracking and registrationof physical features in the multidimensional visualization with thecorresponding features in the multidimensional reconstructed image. Inanother example embodiment, imaging system 12 may orientate amultidimensional reconstructed image with a multidimensionalvisualization of a surgical site by matching tracking coordinates of themultidimensional visualization with coordinates assigned to the imagedata. In some embodiments, an image guidance system (“IGS”) generatesthe coordinates. The image guidance system may include the Medtronic®STEALTHSTATION® or BRAINLAB® KOLIBRI™ system. In one example embodiment,imaging system 12 matches identified physical features of a head (e.g.,brain bone structure or brain blood vessels) to corresponding featuresof the multidimensional reconstructed image. In this example, imagingsystem 12 may match patterns of the blood vessels and brain tissue tothe corresponding patterns and features in the multidimensionalreconstructed image. In some embodiments, imaging system 12 fuses aportion of the multidimensional reconstructed image based on anorientation of the surgical site shown in the displayed visualization.In some embodiments, imaging system 12 fuses a portion of themultidimensional reconstructed image with a displayed visualization byat least one of the following: (a) adjusting transparencies of thevisualization and transparencies of the multidimensional reconstructedimage; (b) smoothing borders between the multidimensional reconstructedimage and the displayed visualization; and (c) adding the portion of themultidimensional reconstructed image to a video signal with placementdetermined by an IGS system; (d) analyzing a video signal to determinethe location and appearance for placing a portion of themultidimensional reconstruction as a graphical image within thevisualization.

In some embodiments, imaging system 12 enables a user to manipulate aninput device to change the orientation of the displayed multidimensionalreconstructed image. In one example embodiment, imaging system 12enables a user to click buttons on the screen to move the position ofthe multidimensional reconstructed image in Cartesian coordinates. Inthis example, the buttons may be appropriately labeled and enable in andout movement along the x, y, and z axes for position, as well as angularalignment known as roll, pitch, and yaw rotation along the x, y, and zaxes. Imaging system 12 may express the position information in anysuitable way, such as cylindrical, (aka polar) coordinates, spherical(aka radial) coordinates, or other coordinate systems.

In some embodiments, imaging system 12 may enable a user to adjust themultidimensional reconstructed image. For example, in one embodiment,imaging system 12 enables a user to grasp a joystick control (e.g., aLOGITECH® SPACE NAVIGATOR®) to adjust the six degrees of freedom whichposition multidimensional reconstructed image relative to themultidimensional visualization.

In some embodiments, imaging system 12 enables a user to move adisplayed multidimensional reconstructed image based on physical changesto a patient's anatomy made during surgery. For example, a surgeon maymove a location of a blood vessel in a multidimensional reconstructedimage by selecting and moving the graphical representation of the bloodvessel on a display of the multidimensional reconstructed image after ablood vessel has physically been moved in the patient. Such aconfiguration enables the imaging system to maintain an accuraterepresentation of a patient's anatomical structure during surgery.

In some alternative embodiments, imaging system 12 displays themultidimensional visualization as superimposed onto a portion of themultidimensional reconstructed image. In one example embodiment, a liveview of a three inch craniotomy is overlayed onto the overall MRI scanof a patient's head.

In some embodiments, imaging system 12 determines which portion of themultidimensional reconstructed image to display based on coordinates ofa selection corresponding to a window. In one example embodiment,imaging system 12 enables a user to generate, place or move a window onthe displayed visualization. In this example, imaging system 12determines which portion of the multidimensional reconstructed imagecorresponds to the received selection based on the generated window.

In some embodiments, imaging system 12 moves the window in response to auser operating with an input device and clicking and dragging thewindow.

In some embodiments, imaging system 12 enables a user to draw the shapeof the window using input device 32.

In some embodiments, imaging system 12 enables a user to set the windowto remain over a specific feature such as a tumor. That is, the windowmay be ‘locked’ onto the feature. Such a configuration enables a user toconstantly know where the feature is located during a surgery.

In one example embodiment, imaging system 12 enables a user to set thedisplay device to constantly display a designated distance (e.g., tenmillimeters) ahead of a currently displayed multidimensionalvisualization. Such a configuration enables a surgeon to graphically seewhat anatomical structure his surgical tools are approaching. In thisexample, as a user moves into a body, imaging system 12 updates thedisplayed multidimensional reconstructed image to display anatomicalstructures that are constantly some specified distance ahead.

In some embodiments, imaging system 12 enables surgeons to use a slideror scroll to visually move through different layers or depths of amultidimensional reconstructed image. Such a configuration enables asurgeon to ‘view’ relatively deep anatomic structures well before asurgeon has physically reached those structures. This configuration alsoenables a surgeon to functionally slide through different layers todetermine what anatomical structure lies between surgical tools and atarget anatomy.

In some embodiments, imaging system 12 enables a user to control of theback-side depth of the multidimensional reconstructed image. Forexample, if a tumor lays close to the cranium, the user may wish to seethe full extent of the tumor in the multidimensional reconstruction, butnot see the dense bone of the cranium which is just behind the tumor. Inthis instance, the user can scroll the rear boundary of themultidimensional reconstruction forward until the cranium bone is nolonger seen in the multidimensional reconstruction

In some embodiments, imaging system 12 automatically displays a portionof a multidimensional reconstructed image having the same depth as thecurrently displayed multidimensional visualization of a target surgicalsite.

In some embodiments, imaging system 12 enables a user to cycle throughportions of multidimensional reconstructed images until a desired depthis displayed. In some instances, imaging system 12 updates a displayedrendered image as a user scrolls through different depths.

In some embodiments, imaging system 12 enables a user to set whichportion of the multidimensional reconstructed image is displayed basedon depth information. For example, in one embodiment where a surgeon iscutting into a brain of a patient, imaging system 12 may receive arequest from a surgeon to display a 3D reconstructed image correspondingto fifteen mm ahead of where the surgeon's scalpel is currently located.In response to such a request, imaging system 12 may determinecoordinates of the currently displayed multidimensional visualizationand then display a portion of the multidimensional reconstructed imagedata that corresponds to fifteen mm ahead of the determined coordinates.In this example, as the surgical tool goes deeper, imaging system 12automatically updates the displayed portion of the multidimensionalreconstructed image to display the portion of the multidimensionalreconstructed image that is fifteen mm ahead of surgery.

In one example embodiment, imaging system 12 enables a user to determinethe thickness of a multidimensional visualization that is underlaid on alive view. For example, the user may not want to see from fifteen mmahead all the way to the other side of the cranium. The surgeon mayprefer to only see from fifteen mm ahead until they reach the site ofthe disease state such as an aneurysm. In this case, imaging system 12may enable the user to select the near and far clipping planes withinthe multidimensional reconstructed image data.

In some embodiments, imaging system 12 filters or enables a user toselect and remove certain types of anatomical structures of amultidimensional reconstructed image. For example, in one embodiment,imaging system 12 enables a user to select to view only bone structures,brain tissue, blood vessels, tumors or aneurisms. Such a configurationenables users to focus the multidimensional reconstructed image ondesired anatomical structures that are important for a surgery.

In some embodiments, imaging system 12 adjusts the transparency of adisplayed multidimensional reconstructed image so that a highlightedanatomical structure may be viewed through different layers of themultidimensional reconstructed image.

In some embodiments, imaging system 12 enables a user to move ormanipulate objects or structure (e.g., a blood vessel or tissue) of adisplayed multidimensional reconstructed image to reflect actualmovement within a patient's anatomy.

In some embodiments, imaging system 12 displays annotations associatedwith a multidimensional reconstructed image. In one example embodiment,before surgery a surgeon may electronically attach notations to or in atleast one pre-operative image. In this example, imaging system 12 storesthe notations and the location of the notation in association with themultidimensional reconstructed image such that the notation is displayedin conjunction with the multidimensional reconstructed image.

In some embodiments, imaging system 12 enables a user to associate asurgical note with a specific portion of image data or a specificportion of a multidimensional reconstructed image. In one exampleembodiment, in response to a user selecting certain portions of adisplayed multidimensional visualization, certain notes which have beenassociated (e.g., attached) with such selected portions are displayedsimultaneously with the displayed portion of the multidimensionalreconstructed image.

In some embodiments, imaging system 12 enables a user to draw a routetrajectory through a multidimensional reconstructed image or a series ofpre-operative images for surgical tools as part of a pre-operative plan.Imaging system 12 may display this route in the displayed portion of themultidimensional reconstructed image in conjunction with amultidimensional visualization of a surgical site.

In some embodiments, imaging system 12 updates which portion of theroute is displayed to coincide with the currently displayedmultidimensional visualization. As a result, imaging system 12 displayssurgical routes in conjunction with a portion of a multidimensionalreconstructed image and a multidimensional visualization of a surgicalsite.

In some embodiments, imaging system 12 may enable a user to control oradjust display characteristics of at least one of the multidimensionalreconstructed image and the multidimensional visualization. Displaycharacteristics may include at least one of color, saturation, hue,luminosity, contrast, brightness, gamma and/or any other displaycharacteristics.

The image data may include any suitable type of data. The image datapreferably corresponds to a surgical area of a patient. The image datamay include at least one of: pre-operative image data; intra-operativeimage data; scan data; any video, image, or data structure that includesmedical information obtained via, for example, a computed tomography(“CT”) scan, a computed tomography agiography (“CT-A”) scan, a magneticresonance imaging (“MRI”) scan, a positron emission tomography (“PET”)scan or any other type of medical scan or imaging; 2D image data; 3Dimage data; and any sequence or video of medical images that conform to,for example, the Digital Imaging and Communications in Medicine(“DICOM”) standard.

The image data may correspond to sequential scans of different depths ofa patient's anatomy.

The image data may be generated or received from any suitable device. Insome embodiments, the image data is received from a medical imagingmachine via a hospital information system. In some embodiments, theimage data is received from another server or another computer.

Imaging system may generate the multidimensional reconstructed imagebased on any suitable method. For example, in one embodiment, imagingsystem 12 generates a 3D reconstructed image based on image data usingvector-based image construction. Vector-based image construction canmerge image characteristics of adjacent two-dimensional (“2D”)pre-operative medical images into 3D shapes and structures. In anotherembodiment imaging system 12 generates a 3D reconstructed image based onvolumetric image data. Volumetric image data may contain multiple valuesdefining the image at each point in space (known as a voxel). The valuesat each voxel may include a density (such as Hounsfield units) or otherintensity parameter. The filters (defined above) may use this voxel datadirectly or a mathematical manipulation of the voxel data such as thegradient of density to define the tissues of interest.

It should be appreciated that the systems and methods disclosed hereinmay provide an effective guide for a user throughout a surgicalprocedure. By enabling a user to see a graphical representation of whatthey are about to cut before they actually cut it, imaging system 12provides the user with a better idea of where they are going and whatthey are to avoid in getting there. The configurations disclosed hereinenable surgeons to ‘view’ anatomic structure below a displayedvisualization before physically reaching that point. Imaging system 12provides surgeons a type of ‘x-ray’ vision into a patent's anatomy inthe form of a multidimensional reconstructed image being displayed toappear as being fused with a real time multidimensional visualization.In other words, such a configuration provides surgeons with a type ofx-ray vision to ‘see’ varying layers of a patient's anatomy in relationto a currently displayed multidimensional visualization without actuallyhaving to physically expose those layers.

It should be appreciated that, in some embodiments, the systems andmethods disclosed herein may enable a surgeon to comfortably visualize asurgical procedure on a display device instead of staring for, in somecases, several hours though the eyepieces of a surgical microscope. Thisis because the real-time visualizations of the systems and methods allowthe surgery to take place in comfortable sitting or standing positionswithout sacrificing a complete and accurate visualization of the targetsurgical field. Traditionally the primary surgeon and any assistantsurgeons have to be physically looking through the microscopeoculars—positioning themselves in rigid and frequently awkwardpositions. By viewing the surgery on a display, a surgeon is free to sitcomfortably and easily move their necks, backs, and shoulders to remainrelaxed and ergonomically situated. These capacities may be ideal for asurgeon and surgical team working long hours. Working such long hoursunder bright lights that generate intense heat in order to visualize thetarget surgical area, as is commonly the case in many known surgicalprocedures, may result in previously unavoidable surgeon discomfort andfatigue. Additionally, it is not uncommon for a surgeon to be wearingseveral layers of clothing along with surgical barriers, includinggloves, face barriers, goggles, hats, and overcoats, to name a few,during a given surgical procedure, further contributing to discomfortand fatigue. Similarly, it is not uncommon for a surgeon to look awayfrom a target surgical site in order to change or to move equipment,glance at other equipment such as IGS and/or patient vital signmonitors, to take a mental break, or to communicate with a surgical teamor students. Upon looking back onto the traditional target surgicalsite, the surgeon would have to wait briefly to allow his eyes to adjustto the normal high intensity lighting in the eyepieces, causing delaysin the procedure. The systems and methods of the present invention mayeliminate this problem by providing a display which fits into the normalvisual filed of the surgeon.

Even further still, the systems and methods described herein allow asurgical team to position themselves in the most appropriate locationfor the surgery, not necessarily where the shadows dictate. Moreover,the systems and methods provide an ideal environment for students toobserve a procedure in comfortable positions especially when used withmultiple screens or with a large display such as a projection screen.

Thus, imaging system 12 provides multidimensional internal guidancerather than just surface guidance. As a result, imaging system 12 mayreduce surgery time, reduce trauma from surgery, and may provide bettersurgery with fewer complications. In addition, the imaging system mayeliminate a need for some visualization probes or cameras in delicate orhard to reach areas.

In some embodiments, the imaging system 12 is a single device. In someembodiments, imaging system 12 is configured to be retrofitted ontoexisting surgical equipment such as surgical microscopes or an opensurgery apparatus. This can be advantageous as the retrofit embodimentsmay be added to existing systems (e.g., microscopes and IGS), allowingexpensive equipment to simply be upgraded as opposed to purchasing anentirely new system. An example imaging system 12 may include variousoptical or electronic magnification systems including stereomicroscopesor may function as an open surgery apparatus utilizing cameras andoverhead visualizations with or without magnification.

FIG. 8 illustrates an example embodiment in which an example imagingsystem is retrofitted onto a surgical microscope. More specifically,surgical microscope 802 is retrofitted with imaging system 804. In thisexample embodiment, imaging system 804 is coupled to first ocular port806 on ocular bridge 808. Further, ocular bridge 808 couples videocamera 810 to a second ocular port (not shown) and binocular eyepiece812 to third ocular port 814. Forth ocular port 816 is available forfurther retrofits to surgical microscope 802. Although surgicalmicroscope 802 has been retrofitted with imaging system 804, it stillretains the use of conventional controls and features such as, but notlimited to, iris adjustment knob 818, first adjustment knob 820, secondadjustment knob 822, illumination control knob 824, and an objectivelens (not shown). Further still, imaging system 804 may send and receiveinformation through signal cable 826.

The imaging system, imaging apparatuses and imaging methods of thepresent invention may be applicable to any form of surgery, such asbrain surgery, spinal surgery, ophthalmologic surgery, cornealtransplants, neurosurgery, orthopedic surgery, ear, nose and throatsurgery, plastics and reconstructive surgery, or general surgery on anytarget structure or tissue.

As discussed above, in some embodiments, touch screen systems may beused to manipulate images and reconstructions. In some embodiments,imaging system 12 enables a user to operate with an input device (e.g.,a 3D mouse such as the SPACE NAVIGATOR®) to position templates, images,and references within the multidimensional reconstructed image. In someembodiments, imaging system 12 includes a foot switch or a lever forpositioning templates, images, and references. Such a configurationenables a user to manipulate multidimensional reconstructed imageswithout taking his or her eyes off of a visualization of a surgicalprocedure, enhancing performance and safety.

In some embodiments, imaging system 12 includes a voice activatedcontrol system. Such a configuration enables a user to control themodification and alignment of multidimensional reconstructed images inconjunction with a multidimensional visualization of a surgical site asif he or she was talking to an assistant or a member of the surgicalteam. The voice activated controls may include a microphone and a seconddata processor or software to interpret the oral voice commands.

In some embodiments, imaging system 12 includes a gesture recognitiondevice configured to enable a user to use gesture commands to controlmultidimensional reconstructed images fused with a visualization of asurgical site. The gesture recognition device may include a camera tomonitor and track the gestures of the controlling user and, optionally,a second data processor or software to interpret the commands.

In some embodiments, imaging system 12 implements camera calibration onone or more photosensors to identify the parameters typically used forimage rectification, camera principal points including: (a) position inx,y,z; (b) rotational orientation in angles phi, theta, psi; (c) focallength and magnification/field of view; and (d) distortion parameterswhich may characterize optical aberrations including any or all of thefollowing: defocus, piston, tilt, shear, astigmatism, coma, or otherhigher order aberrations or chromatic aberrations. Once characterized,imaging system 12 may apply corrections to the video signal whichresults in an orthoscopic or rectilinear visualization of the surgery.

In some embodiments, imaging system 12 employs camera calibrationparameters to apply the optical distortion that is present in the videosignal to the reconstruction such that the final rendering of thepre-operative data reflects the same distortion and aberrations presentin the video signal. By matching the lighting, geometry, and distortionparameters between the live video signal and the 3D reconstruction, thegraphical multidimensional reconstructed image will most precisely matchthe live view of the surgical sight and provide the surgeon with themost accurate navigational guidance during the procedure.

FIG. 9 is a block diagram of an example data architecture 900. In thisexample embodiment, interface data 902, administrative data 904, anddata 906 interact with each other, for example, based on user commandsor requests. Interface data 902, administrative data 904, and data 906may be stored on any suitable storage medium (e.g., database system 302and/or server 14). It should be appreciated that different types of datamay use different data formats, storage mechanisms, etc. Further,various applications may be associated with processing interface data902, administrative data 904, and data 906. Various other or differenttypes of data may be included in the example data architecture 900.

Interface data 902 may include input and output data of various kinds.For example, input data may include mouse click data, scrolling data,hover data, keyboard data, touch screen data, voice recognition data,etc., while output data may include image data, text data, video data,audio data, etc. Interface data 902 may include formatting, userinterface options, links or access to other websites or applications,and the like. Interface data 902 may include applications used toprovide or monitor interface activities and handle input and outputdata.

Administrative data 904 may include data and applications regarding useraccounts. For example, administrative data 904 may include informationused for updating accounts, such as creating or modifying user accountsand/or host accounts. Further, administrative data 904 may includeaccess data and/or security data. Administrative data 904 may include aterms of service agreement. Administrative data 904 may interact withinterface data 902 in various manners, providing interface data 902 withadministrative features, such as implementing a user login and the like.

Data 906 may include, for example, multidimensional visualization data908, multidimensional reconstructed image data 910, selection data 912,window data 914, and image data 916.

Multidimensional visualization data 908 may include data representativeof at least one of: a surgical site, a 3D visualization of a surgicalsite, a 2D visualization of a surgical site, and real time data.

Multidimensional reconstructed image data 910 may include datarepresentative of at least one of: feature data, brain tumor data, braintissue data, bone structure data, aneurysm data, blood vessel data,vertebrate data, coordinate data, depth data, distance data, andtransparency data.

Selection data 912 may include data representative of at least one of: aportion of a multidimensional visualization.

Window data 914 may include data representative of position data.

Image data 916 may include data representative of at least one of:pre-operative image data, intra operative image data, medical scan data,and image slice data.

The imaging system may include components of Applicant's TrueVisionSystems, Inc. real-time 3D HD visualization systems described inApplicant's co-pending U.S. applications: Ser. No. 11/256,497 entitled“Stereoscopic Image Acquisition Device,” filed Oct. 21, 2005; Ser. No.11/668,400 entitled “Stereoscopic Electronic Microscope Workstation,”filed Jan. 29, 2007; Ser. No. 11/668,420 entitled “StereoscopicElectronic Microscope Workstation,” filed Jan. 29, 2007; Ser. No.11/739,042 entitled “Stereoscopic Display Cart and System,” filed Apr.23, 2007; and Ser. No. 61/042,606, entitled “Apparatus and Methods forPerforming Enhanced Visually Directed Procedures Under Low Ambient LightConditions,” filed Apr. 4, 2008, all of which are fully incorporatedherein by reference as if part of this specification.

“Realtime” as used herein generally refers to the updating ofinformation at essentially the same rate as the data is received. Morespecifically, “realtime” is intended to mean that the image data isacquired, processed, and transmitted from the photosensor of thevisualization generation system at a high enough data rate and at a lowenough time delay that when the data is displayed, objects presented inthe visualization move smoothly without user-noticeable judder, latencyor lag. Typically, this occurs when new images are acquired, processed,and transmitted at a rate of at least about 30 frames per second (“fps”)and displayed at a rate of at least about 60 fps and when the combinedprocessing of the video signal has no more than about 1/10^(th) of asecond of delay.

In some embodiments, new images are acquired, processed, and transmittedat a rate of at least about 24 fps, about 30 fps, about 35 fps, about 40fps, about 50 fps, about 60 fps, about 70 fps, about 80 fps, about 90fps or about 120 fps. Also, new images are displayed at a rate of atleast about 60 fps, about 70 fps, about 80 fps, about 90 fps or about120 fps. The signal processing may have no more than about 1/20^(th)second of delay, about 1/30^(th) second of delay, about 1/50^(th) secondof delay, about 1/90^(th) second of delay, about 1/120^(th) second ofdelay, about 1/500^(th) second of delay, or about 1/1000^(th) seconddelay or more.

The term “high definition” or “HD” as used herein may encompass a videosignal having a resolution of at least 960 lines by 720 lines and togenerally have a higher resolution than a standard definition (SD)video. For purposes of the present disclosure, this may be accomplishedwith display resolutions of 1280 lines by 720 lines (720p and 720i) or1920 lines by 1080 lines (1080p or 1080i). In contrast, standarddefinition (SD) video typically has a resolution of 640 lines by 480lines (480i or 480p) or less. It is however, within the scope of thepresent disclosure that the multidimensional visualization may be in SD,though HD is preferred. Further implementations using 4 k displays witha resolution up to 4096 by 2160, 8 k displays with a resolution up to7680 by 4320 are within the scope of the invention.

Unless otherwise indicated, all numbers expressing quantities ofingredients, properties such as molecular weight, reaction conditions,and so forth used in the specification and claims are to be understoodas being modified in all instances by the term “about.” Accordingly,unless indicated to the contrary, the numerical parameters set forth inthe specification and attached claims are approximations that may varydepending upon the desired properties sought to be obtained by thepresent disclosure. At the very least, and not as an attempt to limitthe application of the doctrine of equivalents to the scope of theclaims, each numerical parameter should at least be construed in lightof the number of reported significant digits and by applying ordinaryrounding techniques. Notwithstanding that the numerical ranges andparameters setting forth the broad scope of the disclosure areapproximations, the numerical values set forth in the specific examplesare reported as precisely as possible. Any numerical value, however,inherently contains certain errors necessarily resulting from thestandard deviation found in their respective testing measurements.

The terms “a,” “an,” “the” and similar referents used in the context ofdescribing the disclosure (especially in the context of the followingclaims) are to be construed to cover both the singular and the plural,unless otherwise indicated herein or clearly contradicted by context.Recitation of ranges of values herein is merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range. Unless otherwise indicated herein, eachindividual value is incorporated into the specification as if it wereindividually recited herein. All methods described herein may beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein isintended merely to better illuminate the disclosure and does not pose alimitation on the scope of the disclosure otherwise claimed. No languagein the specification should be construed as indicating any non-claimedelement essential to the practice of the disclosure.

Groupings of alternative elements or embodiments of the disclosuredisclosed herein are not to be construed as limitations. Each groupmember may be referred to and claimed individually or in any combinationwith other members of the group or other elements found herein. It isanticipated that at least one member of a group may be included in, ordeleted from, a group for reasons of convenience and/or patentability.When any such inclusion or deletion occurs, the specification is deemedto contain the group as modified thus fulfilling the written descriptionof all Markush groups used in the appended claims.

Certain embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the disclosure. Ofcourse, variations on these described embodiments will become apparentto those of ordinary skill in the art upon reading the foregoingdescription. The inventors expect skilled artisans to employ suchvariations as appropriate, and the inventors intend for the disclosureto be practiced otherwise than specifically described herein.Accordingly, this disclosure includes all modifications and equivalentsof the subject matter recited in the claims appended hereto as permittedby applicable law. Moreover, any combination of the above-describedelements in all possible variations thereof is encompassed by thedisclosure unless otherwise indicated herein or otherwise clearlycontradicted by context.

Furthermore, numerous references have been made to patents and printedpublications throughout this specification. Each of the above-citedreferences and printed publications are individually incorporated hereinby reference in their entirety.

Specific embodiments disclosed herein may be further limited in theclaims using consisting of or and consisting essentially of language.When used in the claims, whether as filed or added per amendment, thetransition term “consisting of” excludes any element, step, oringredient not specified in the claims. The transition term “consistingessentially of” limits the scope of a claim to the specified materialsor steps and those that do not materially affect the basic and novelcharacteristic(s). Embodiments of the disclosure so claimed areinherently or expressly described and enabled herein.

In closing, it is to be understood that the embodiments of thedisclosure disclosed herein are illustrative of the principles of thepresent disclosure. Other modifications that may be employed are withinthe scope of the disclosure. Thus, by way of example, but not oflimitation, alternative configurations of the present disclosure may beutilized in accordance with the teachings herein. Accordingly, thepresent disclosure is not limited to that precisely as shown anddescribed.

The invention is claimed as follows:
 1. An imaging system comprising: aprocessor; an input device operatively connected to the processor; adisplay device operatively connected to the processor; and a memorydevice storing instructions which when executed by the processor, causethe processor to operate with the input device, the display device andthe memory device to: (a) display a multidimensional visualization of asurgical site; (b) receive a selection corresponding to a portion of thedisplayed multidimensional visualization of the surgical site; and (c)at the selected portion of the multidimensional visualization, display aportion of a multidimensional reconstructed image which corresponds tothe selected multidimensional visualization, the displayed portion ofthe multidimensional reconstructed image being fused with the displayedmultidimensional visualization.
 2. The imaging system of claim 1,wherein the multidimensional visualization is three dimensional anddisplayed in real time.
 3. The imaging system of claim 1, wherein themultidimensional reconstructed image is: (a) three dimensional; and (b)generated based on image data captured by a medical imaging machine. 4.The imaging system of claim 3, wherein the image data includes twodimensional pre-operative image slice data.
 5. The imaging system ofclaim 1, wherein the selected portion of the multidimensionalvisualization is represented as a window.
 6. The imaging system of claim5, wherein the window has a shape selected from the group consisting ofcircular shape, square shape, rectangular shape and anatomic shape. 7.The imaging system of claim 5, wherein the instructions, when executedby the processor, cause the processor to move the position of the windowin response to the window being clicked and dragged.
 8. The imagingsystem of claim 1, wherein the multidimensional reconstructed imageincludes a first feature and a second feature.
 9. The imaging system ofclaim 8, wherein: (a) the first feature corresponds to a tumor; and (b)the second feature corresponds to bone structure.
 10. The imaging systemof claim 8, wherein the instructions, when executed by the processor,highlight the first feature.
 11. The imaging system of claim 8, whereinthe instructions, when executed by the processor, filter out the firstfeature.
 12. The imaging system of claim 1, wherein the instructions,when executed by the processor, cause the processor to: (a) receive arequest for a second portion of the multidimensional reconstructed imagehaving a specific depth; and (b) display the requested second portion ofthe multidimensional reconstructed image having the specific depth. 13.The imaging system of claim 1, wherein the instructions, when executedby the processor, cause the processor to: (a) for the displayedmultidimensional reconstructed image, receive a request for a depthsetting to correspond to a depth that is a specific distance belowcoordinates of the displayed multidimensional visualization; and (b)display the portion of the multidimensional reconstructed image having adepth that is a the specific distance below the coordinates of thedisplayed multidimensional visualization.
 14. The imaging system ofclaim 1, wherein the instructions, when executed by the processor, causethe processor to fuse the displayed portion of the multidimensionalreconstructed image by identifying first features of themultidimensional visualization and aligning the first features tocorrespond to second features of the multidimensional reconstructedimage.
 15. The imaging system of claim 1, wherein the instructions, whenexecuted by the processor, cause the processor to integrate themultidimensional reconstructed image with the multidimensionalvisualization by: (a) increasing a transparency of the selected portionof the multidimensional visualization; and (b) applying a smoothingfunction to a boundary between the multidimensional reconstructed imageand the multidimensional visualization.
 16. A method of operating animaging system, the method comprising: (a) causing a processor toexecute instructions to operate with a display device to display amultidimensional visualization of a surgical site; (b) causing theprocessor to execute the instructions to receive a selectioncorresponding to a portion of the displayed multidimensionalvisualization of the surgical site; and (c) causing the processor toexecute the instructions to operate with the display device to, at theselected portion of the multidimensional visualization, display aportion of a multidimensional reconstructed image which corresponds tothe selected multidimensional visualization, the displayed portion ofthe multidimensional reconstructed image being fused with the displayedmultidimensional visualization.
 17. The method of claim 16, whichincludes causing the processor to execute the instructions to operatewith an input device to move the position of a window in response to thewindow being clicked and dragged.
 18. The method of claim 16, whichincludes causing the processor to execute the instructions to operatewith an input device to highlight a feature of the multidimensionalreconstructed image.
 19. The method of claim 16, which includes causingthe processor to execute the instructions to operate with an inputdevice to filter out a feature of the multidimensional reconstructedimage.
 20. A non-transitory computer readable medium storinginstructions structured to cause an imaging system to: (a) display amultidimensional visualization of a surgical site; (b) receive aselection corresponding to a portion of the displayed multidimensionalvisualization of the surgical site; and (c) at the selected portion ofthe multidimensional visualization, display a portion of amultidimensional reconstructed image which corresponds to the selectedmultidimensional visualization, the displayed portion of themultidimensional reconstructed image being fused with the displayedmultidimensional visualization.